- Representation
- AISystems are the main class users interact with in RAI, they capture key information about an AI.
- This information is passed during construction and includes a name, a task type, a MetaDatabase, a Dataset and a Model.
- Interaction
- AISystems make it simple to run computations and get metric values, and are needed to run an Analysis and run Certifications.
- After making an AISystem, users can use the compute function to generate all relevant metrics related to their model and dataset.
- The Model, Task Type and MetaDatabase are RAI classes which provide critical information to the AISystem, allowing it to determine which metrics and analyses are relevant.
- After computing metrics, users can get retrieve metric values using the get_metric_values function.
- When provided a network's functions to generate predictions or values, AISystems can use models and run evaluations without requiring user involvement.
- You can provide custom expressions or functions as metrics that will be evaluated for every measurement.
Example:
AI_sys file example
Important
Rai utilization: RAI will utilize AISystem to compute and determine which metrics are relevant (e.g. of the chosen model design).
- Representation
- Certificates allow users to quickly and easily define and communicate standards for AISystems in different domains and tasks.
- Once a certificate has been added to an AISystem, the AISystem can quickly and easily evaluate whether or not it meets the standards of the certificate allowing for quick yet robust evaluation.
- Certificates are written in JSON and can contain logical and relational operators, with the ability to retrieve any metric associated with an AISystem.
- Interaction
- Custom Certificates can be added to an AISystem by passing in a filepath to the certificate file while initializing the AISystem.
- Certificates will be evaluated when the AISystem calls its compute function.
- Certificates can be retrieved by calling the get_certificate_values function on the AISystem.
Example:
- display_name: Adversarial Bound Bronze Certification.
- description: Certifies whether or not the agent is robust against adversarial attacks.
Certi file example
Important
Rai utilization: RAI will carry out detailed analyses (e.g. of the chosen model design) and tests (e.g. robustness, bias, explainability) and define a certification that have Accuracy features.
- Representation
- Each Metric comes with metadata, including its name and description, as well as a function to compute the metric.
- Metrics are grouped into MetricGroups, which are collections of Metrics with similar compatibility and functionality.
- AISystems access metrics through MetricManagers which are responsible for checking compatibility between MetricGroups and AISystems, as well as computing and retrieving specific Metric values.
- MetricManagers are automatically created and managed by AISystems and are the key to running Metrics and retrieving their values.
- Interaction
- Interaction with Metrics are done through MetricManagers.
- MetricManagers are capable of quickly finding all MetricGroups compatible with an AISystem.
- RAI ensures that dependencies between Metrics are satisfied with no circular dependency issues.
- Functionality is provided to search for specific Metrics based on Metric Name, Metric Group Name, Category, and Tags.
- Metrics are compatible with both whole and batched data.
Example:
Metric file example
Important
Rai utilization: RAI will utilize Metrics to monitor and measures the performance of a model (during training and testing).
- Representation
- While metrics are typically general and simple to calculate, Analyses are finegrained evaluations to run on specific AISystems.
- Analyses provide a way for users to quickly and easily run complex experiments compatible with their model, with built in visualizations.
- Analyses are easy to create allowing users to quickly and easily make their own custom Analyses for their specific needs using any attribute of the AISystem.
- Interaction
- Analyses are managed by the AnalysisManger and are given access to the AISystem and Dataset through the RaiDB class.
- Similar to MetricManagers, AnalysisManagers check compatibility between Analyses and AISystems and handle the computation of Analyses.
- Running specific analyses is done through the run_analysis function.
Example:
Analysis file example
Important
Rai utilization: RAI will carry out detailed analyses and automates report generation and makes data easy to understand.